Feat : Add aggregation from kube service endpoints feature in metrics API scaler #6565

julianguinard · 2025-02-24T16:16:07Z

TL:DR

Add the ability for metrics-api scaler to get metrics from all endpoint targets of a kubernetes API service (aggregateFromKubeServiceEndpoints: true metadata) and aggregate them using average, sum, min or max aggregation functions (aggregationType metadata), which is a handy feature in an environment where one didn't set up a metric aggregator/scraping stack (i.e prometheus), or simply doesn't want to use their monitoring stack to fetch and serve metrics from customers workload in their own kubernetes clusters, and leave the metrics API's reponsability up to the customer

This PR comes from an issue met when metrics-api scaler targets an internal kubernetes service, such as the one from metrics_api E2E tests but, unlike it, has more than 1 replica serving it, leading to inconsistent HPA average calculation

The more a kubernetes service used by metrics API scaler has pods as targets, the less likely it is for the metrics API scaler to get all metrics from all these pods, leading to inconsistent HPA average metric computation and eventually to scaling issues. Especially in configurations where kubernetes services have "sticky" configurations

Below are screenshots of the modified E2E metrics_api_test.go, commenting out aggregateFromKubeServiceEndpoints: true addition to metadata, being executed & failing because scale in never occurs as every metric from all 10 replicas behind metrics server deployment are not all well taken into account

logs from keda-operator show same value being returned to metrics-api scaler by the kubernetes service (16)

As a consequence, test eventually fails because scaled deployment is unable to scale out within 3 minutes timeframe

Uncommenting aggregateFromKubeServiceEndpoints: true addition to metadata results in metric aggregation mode and same test passes this time

Checklist

When introducing a new scaler, I agree with the scaling governance policy
I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Helm PR : kedacore/charts#739
Docs PR : kedacore/keda-docs#1541

tests/scalers/metrics_api/metrics_api_test.go

zroubalik · 2025-03-05T13:46:54Z

/run-e2e metrics_api
Update: You can check the progress here

julianguinard · 2025-03-06T09:06:47Z

/run-e2e metrics_api Update: You can check the progress here

@zroubalik thanks for triggering E2E, which ran fine here

julianguinard · 2025-03-11T15:32:03Z

@zroubalik FYI I added this commit to make E2E more resilient on clusters where launching all metrics servers replicas take a longer time than expected to start : we now sleep 1 second & try 4 more times to validate expected pods before failing

It also fixes formatting of each endpoint URL to reach for aggregating metrics (double "/" in path, did not trigger errors in E2E because server ran by ghcr.io/kedacore/tests-metrics-api seems to cope with it anyway)

JorTurFer · 2025-03-14T22:15:39Z

/run-e2e metrics_api
Update: You can check the progress here

julianguinard · 2025-03-17T08:52:50Z

/run-e2e metrics_api Update: You can check the progress here

@JorTurFer thanks for E2E trigger which all ran fine

waiting for reviews & opinions on this now :)

pkg/scalers/metrics_api_scaler.go

…rics from all kubernetes service endpoints fail for whatever reason see kedacore#6565 (comment) Signed-off-by: julian GUINARD <[email protected]>

JorTurFer · 2025-03-24T10:09:35Z

/run-e2e metrics_api
Update: You can check the progress here

…rics from all kubernetes service endpoints fail for whatever reason see kedacore#6565 (comment) Signed-off-by: julian GUINARD <[email protected]>

JorTurFer · 2025-03-24T12:50:33Z

/run-e2e metrics_api
Update: You can check the progress here

JorTurFer

Nice job! I've kept some implementation ideas inline

pkg/scalers/metrics_api_scaler.go

julianguinard · 2025-10-01T07:44:50Z

Repushed for changelog conflict solving

rickbrouwer · 2025-10-01T08:08:18Z

/run-e2e metrics
Update: You can check the progress here

wozniakjan

this looks super usefull, few items regarding EndpointSlice usage below

pkg/scalers/metrics_api_scaler.go

- kedacore#6565 (comment) - kedacore#6565 (comment) Signed-off-by: julian GUINARD <[email protected]>

julianguinard · 2025-10-08T08:13:42Z

this looks super usefull, few items regarding EndpointSlice usage below

Hello @wozniakjan and thanks for your feedback ❤️
I adapted EndpointSlice usage according to tour suggestions. In the iteration, we now discard any unready endpoint or endpoint adresses which have already been registered
feel free to review it again!

pkg/scalers/metrics_api_scaler.go

…scaler - make generate-scalers-schema Signed-off-by: julian GUINARD <[email protected]>

- kedacore#6565 (comment) - kedacore#6565 (comment) Signed-off-by: julian GUINARD <[email protected]>

Co-authored-by: Jan Wozniak <[email protected]> Signed-off-by: julianguinard <[email protected]>

rickbrouwer · 2025-10-20T16:12:33Z

/run-e2e metrics_api
Update: You can check the progress here

julianguinard · 2025-10-21T15:01:19Z

E2E all green. @wozniakjan feel free to have a third look at it whenever you wish :)

julianguinard · 2025-10-23T08:29:50Z

Thanks for the mege @wozniakjan !
we may also merge twin doc & helm chart PRs
kedacore/keda-docs#1617
kedacore/charts#739

… API scaler (kedacore#6565) * - Add aggregation from kube service endpoints feature in metrics API scaler - make generate-scalers-schema Signed-off-by: julian GUINARD <[email protected]> * apply suggestions : - kedacore#6565 (comment) - kedacore#6565 (comment) Signed-off-by: julian GUINARD <[email protected]> * Update pkg/scalers/metrics_api_scaler.go Co-authored-by: Jan Wozniak <[email protected]> Signed-off-by: julianguinard <[email protected]> --------- Signed-off-by: julian GUINARD <[email protected]> Signed-off-by: julianguinard <[email protected]> Co-authored-by: Jan Wozniak <[email protected]> Signed-off-by: Dmitriy Altuhov <[email protected]>

… API scaler (kedacore#6565) * - Add aggregation from kube service endpoints feature in metrics API scaler - make generate-scalers-schema Signed-off-by: julian GUINARD <[email protected]> * apply suggestions : - kedacore#6565 (comment) - kedacore#6565 (comment) Signed-off-by: julian GUINARD <[email protected]> * Update pkg/scalers/metrics_api_scaler.go Co-authored-by: Jan Wozniak <[email protected]> Signed-off-by: julianguinard <[email protected]> --------- Signed-off-by: julian GUINARD <[email protected]> Signed-off-by: julianguinard <[email protected]> Co-authored-by: Jan Wozniak <[email protected]>

the final semaphore acquisition was still missing, which resulted in loop not waiting until all metrics were gathered from each metrics API server Signed-off-by: julian GUINARD <[email protected]>

the final semaphore acquisition was still missing, which resulted in loop not waiting until all metrics were gathered from each metrics API server Signed-off-by: julian GUINARD <[email protected]> Signed-off-by: Dima Altukhov <[email protected]>

semgrep-app bot reviewed Feb 24, 2025

View reviewed changes

tests/scalers/metrics_api/metrics_api_test.go Outdated Show resolved Hide resolved

semgrep-app bot reviewed Feb 24, 2025

View reviewed changes

tests/scalers/metrics_api/metrics_api_test.go Outdated Show resolved Hide resolved

This was referenced Feb 24, 2025

update metrics-api scaler doc related to fetching metrics from kubernetes service kedacore/keda-docs#1541

Closed

add get/list/watch operations rights on endpoints to keda-operator kedacore/charts#739

Merged

julianguinard force-pushed the add-metrics-api-aggregation-from-kube-service-feature branch 2 times, most recently from 742e19d to be170fd Compare February 25, 2025 09:57

julianguinard marked this pull request as ready for review February 25, 2025 10:31

julianguinard requested a review from a team as a code owner February 25, 2025 10:31

julianguinard changed the title ~~Add aggregation from kube service endpoints feature in metrics API scaler~~ Feat : Add aggregation from kube service endpoints feature in metrics API scaler Feb 25, 2025

julianguinard force-pushed the add-metrics-api-aggregation-from-kube-service-feature branch 2 times, most recently from 172e504 to 61ebd1f Compare March 11, 2025 15:02

JorTurFer reviewed Mar 17, 2025

View reviewed changes

pkg/scalers/metrics_api_scaler.go Show resolved Hide resolved

julianguinard requested a review from JorTurFer March 19, 2025 10:31

julianguinard force-pushed the add-metrics-api-aggregation-from-kube-service-feature branch from 1c33197 to 25f9b0d Compare March 24, 2025 10:37

julianguinard force-pushed the add-metrics-api-aggregation-from-kube-service-feature branch from 215053a to 908f8c3 Compare March 24, 2025 10:43

julianguinard force-pushed the add-metrics-api-aggregation-from-kube-service-feature branch from 908f8c3 to c6ca0e4 Compare March 24, 2025 10:44

JorTurFer reviewed Apr 26, 2025

View reviewed changes

pkg/scalers/metrics_api_scaler.go Show resolved Hide resolved

pkg/scalers/metrics_api_scaler.go Outdated Show resolved Hide resolved

rickbrouwer approved these changes Sep 29, 2025

View reviewed changes

keda-automation requested review from a team October 1, 2025 07:40

wozniakjan reviewed Oct 1, 2025

View reviewed changes

pkg/scalers/metrics_api_scaler.go Outdated Show resolved Hide resolved

pkg/scalers/metrics_api_scaler.go Show resolved Hide resolved

julianguinard added a commit to julianguinard/keda that referenced this pull request Oct 8, 2025

apply suggestions :

5bc7805

- kedacore#6565 (comment) - kedacore#6565 (comment) Signed-off-by: julian GUINARD <[email protected]>

julianguinard force-pushed the add-metrics-api-aggregation-from-kube-service-feature branch from d0497e5 to 5bc7805 Compare October 8, 2025 07:51

keda-automation requested a review from a team October 8, 2025 07:51

wozniakjan reviewed Oct 8, 2025

View reviewed changes

pkg/scalers/metrics_api_scaler.go Outdated Show resolved Hide resolved

keda-automation requested a review from a team October 8, 2025 12:05

julianguinard and others added 3 commits October 8, 2025 14:26

- Add aggregation from kube service endpoints feature in metrics API …

23bb4b6

…scaler - make generate-scalers-schema Signed-off-by: julian GUINARD <[email protected]>

apply suggestions :

3255464

- kedacore#6565 (comment) - kedacore#6565 (comment) Signed-off-by: julian GUINARD <[email protected]>

Update pkg/scalers/metrics_api_scaler.go

0a5d42c

Co-authored-by: Jan Wozniak <[email protected]> Signed-off-by: julianguinard <[email protected]>

julianguinard force-pushed the add-metrics-api-aggregation-from-kube-service-feature branch from 881798c to 0a5d42c Compare October 8, 2025 12:27

wozniakjan approved these changes Oct 21, 2025

View reviewed changes

wozniakjan merged commit 8471b65 into kedacore:main Oct 21, 2025
24 checks passed

julianguinard deleted the add-metrics-api-aggregation-from-kube-service-feature branch October 22, 2025 09:08

julianguinard mentioned this pull request Jan 5, 2026

Bugfix - Metrics API / aggregate from kubernetes service endpoint slices : add missing final semaphore acquisition #7370

Merged

2 tasks

Feat : Add aggregation from kube service endpoints feature in metrics API scaler #6565

Feat : Add aggregation from kube service endpoints feature in metrics API scaler #6565

Uh oh!

Conversation

julianguinard commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

zroubalik commented Mar 5, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

julianguinard commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

julianguinard commented Mar 11, 2025

Uh oh!

JorTurFer commented Mar 14, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

julianguinard commented Mar 17, 2025

Uh oh!

Uh oh!

JorTurFer commented Mar 24, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JorTurFer commented Mar 24, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JorTurFer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

julianguinard commented Oct 1, 2025

Uh oh!

rickbrouwer commented Oct 1, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wozniakjan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

julianguinard commented Oct 8, 2025

Uh oh!

Uh oh!

rickbrouwer commented Oct 20, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

julianguinard commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

julianguinard commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

julianguinard commented Feb 24, 2025 •

edited

Loading

zroubalik commented Mar 5, 2025 •

edited by github-actions bot

Loading

julianguinard commented Mar 6, 2025 •

edited

Loading

JorTurFer commented Mar 14, 2025 •

edited by github-actions bot

Loading

JorTurFer commented Mar 24, 2025 •

edited by github-actions bot

Loading

JorTurFer commented Mar 24, 2025 •

edited by github-actions bot

Loading

rickbrouwer commented Oct 1, 2025 •

edited by github-actions bot

Loading

rickbrouwer commented Oct 20, 2025 •

edited by github-actions bot

Loading

julianguinard commented Oct 21, 2025 •

edited

Loading